Automatic image inpainting

ABSTRACT

The technical problem of removing an object depicted in a selected region of an image to create a natural-looking edited image is addressed by providing systems, methods, and computer-readable storage media to perform automatic image inpainting. The method includes replacing the selected region using a color mask. A color mask can be generated using a mean color of pixels from a portion of the image that is distinct from and outside of the selected region.

PRIORITY CLAIMS

This application is a continuation of and claims the benefit of priorityof U.S. patent application Ser. No. 16/829,720, filed on Mar. 25, 2020,which is a continuation of and claims the benefit of priority of U.S.patent application Ser. No. 16/387,110, filed on Apr. 17, 2019, which isa continuation of and claims the benefit of priority of U.S. patentapplication Ser. No. 16/122,639, filed on Sep. 5, 2018, which is acontinuation of and claims the benefit of priority of U.S. patentapplication Ser. No. 15/448,216, filed on Mar. 2, 2017, each of which ishereby incorporated by reference herein in their entireties.

TECHNICAL FIELD

The present disclosure generally relates to the technical field ofspecial-purpose machines for performing image inpainting, includingcomputerized variants of such special-purpose machines and improvementsto such variants, and to the technologies by which such special-purposemachines become improved compared to other special-purpose machines thatperform image inpainting. In particular, the present disclosureaddresses systems and methods for automatic image inpainting using localpatch statistics.

BACKGROUND

As the popularity of social networking grows, the number of digitalimages generated and shared using social networks grows as well. Priorto sharing such digital images on social networks, users may wish toremove certain objects depicted in the images or other undesirableelements of the images. Among other things, embodiments of the presentdisclosure help users perform edits to digital images, such as removingcertain regions of an image and filling these regions with otherportions of the image to create a natural-looking edited image.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. Some embodiments are illustrated by way of example, and notlimitation, in the figures of the accompanying drawings.

FIG. 1 is a block diagram showing an example messaging system forexchanging data (e.g., messages and associated content) over a network,according to some embodiments.

FIG. 2 is block diagram illustrating further details regarding themessaging system, according to some embodiments.

FIG. 3 is a schematic diagram illustrating data which may be stored in adatabase of the messaging system, according to some embodiments.

FIG. 4 is a block diagram illustrating functional components of an imageprocessing system that forms part of the messaging system, according tosome example embodiments.

FIGS. 5-8 are flow charts illustrating operations of the imageprocessing system in performing an example method for digital imageediting, according to some embodiments.

FIGS. 9A and 9B are interface diagrams illustrating aspects of userinterfaces provided by the messaging system, according to someembodiments.

FIG. 10 is a block diagram illustrating a representative softwarearchitecture, which may be used in conjunction with various hardwarearchitectures herein described.

FIG. 11 is a block diagram illustrating components of a machine,according to some exemplary embodiments, able to read instructions froma machine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.

DETAILED DESCRIPTION

The description that follows includes systems, methods, techniques,instruction sequences, and computing machine program products thatembody illustrative embodiments of the disclosure. In the followingdescription, for the purposes of explanation, numerous specific detailsare set forth in order to provide an understanding of variousembodiments of the inventive subject matter. It will be evident,however, to those skilled in the art, that embodiments of the inventivesubject matter may be practiced without these specific details. Ingeneral, well-known instruction instances, protocols, structures, andtechniques are not necessarily shown in detail.

Aspects of the present disclosure include systems, methods, techniques,instruction sequences, and computing machine program products that allowa user to select an object, region, or other element in an originalimage to be removed and replaced using other portions (e.g. background)of the image, thereby making the resulting edited image appear morenatural. As an example, a user may take a picture of two people, andselect one of the two people for removal from the picture. Uponreceiving an indication of the selection of person to be removed, thesystem removes the selected person from the image and inpaints (e.g.,fills) the missing region (e.g., the region with the person removed)using portions of the picture near the missing region. The result ofthis process is an edited picture of a single person that appearsnatural despite omitting the second person that was in the originalimage.

Consistent with some embodiments, the system determines a local regionfor a user-selected region that includes the object or other element theuser seeks to remove. The systems selects the portions of the image toinpaint (e.g., fill) the user-selected region from the local region.More specifically, the system identifies patch matches within the localregion (e.g., two identical image patches that each comprise one or morepixels) and uses a portion of the identified patch matches to inpaintthe user-selected region.

As part of this process, the system computes local patch matchstatistics that comprise patch offsets for the identified patch matches.The patch offsets include a distance (e.g., represented in two spatialdimensions) between patch matches. The system uses the patch matchstatistics to build a spatial histogram. To fill the user selectedregion, a pixel-level graph cut algorithm may be applied, where thelabel of each pixel in the user-selected region corresponds to apossible (x, y) offset in the histogram. By computing patch matchstatistics only in the local region rather than the entire image, thesystem achieves a faster runtime speed compared to other methods thatuse patch match statistics from the entire image. Additionally, bylimiting the computation of patch matches to the local region, thesystem employs improved techniques for image inpainting that overcomedifficulties encountered by conventional methodologies in processingcomplex backgrounds or cluttered scenes.

To incorporate sufficient patch match statistics when the user-selectedregion is close to the image border, the system may pad the originalimage by a predefined padding size (e.g., by appending a reflection ofthe outer portion of the image to the image border), and enlarge thelocal region by the predefined padding size to allow sufficient patchmatch statistics to be calculated. The system may further scale (e.g.,resize) the user-selected region to a predetermined size (e.g., 100×75pixels) before filling the region with the portion of the identifiedpatch matches.

Additionally, in some instances, the techniques used to fill theuser-selected region with portions of the patch matches may createstrong edges (e.g., pronounced image brightness discontinuities) in theinpainted image. To compensate for this, the system appliesedge-preserving filtering techniques to blur insignificant edges. Thesystem may then identify the strong edges from the filtered image togenerate an edge map. The system may further dilate the edge map togenerate a binary mask for possible strong edge pixels. The system alsoapplies blurring techniques (e.g., Gaussian blur) on the inpaintingresult to generate a blurred version of the inpainted image. The systemmay then blend together the original inpainted image with the blurredversion by applying blending techniques (e.g., Laplacian blending) withthe binary mask. In this way, the system may produce high-qualitynatural-looking inpainted images while being optimized for runtime speed(especially on mobile configurations).

FIG. 1 is a block diagram showing an example messaging system 100 forexchanging data (e.g., messages and associated content) over a network.The messaging system 100 includes multiple client devices 102, each ofwhich hosts a number of applications including a messaging clientapplication 104. Each messaging client application 104 iscommunicatively coupled to other instances of the messaging clientapplication 104 and a messaging server system 108 via a network 106(e.g., the Internet). As used herein, the term “client device” may referto any machine that interfaces with a communications network (such asthe network 106) to obtain resources from one or more server systems orother client devices. A client device may be, but is not limited to, amobile phone, desktop computer, laptop, portable digital assistant(PDA), smart phone, tablet, ultra book, netbook, laptop, multi-processorsystem, microprocessor-based or programmable consumer electronicssystem, game console, set-top box, or any other communication devicethat a user may use to access a network.

In the example shown in FIG. 1 , each messaging client application 104is able to communicate and exchange data with another messaging clientapplication 104 and with the messaging server system 108 via the network106. The data exchanged between the messaging client applications 104,and between a messaging client application 104 and the messaging serversystem 108, includes functions (e.g., commands to invoke functions) aswell as payload data (e.g., text, audio, video, or other multimediadata).

The network 106 may include, or operate in conjunction with, an ad hocnetwork, an intranet, an extranet, a virtual private network (VPN), alocal area network (LAN), a wireless LAN (WLAN), a wide area network(WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), theInternet, a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a plain old telephone service (POTS) network,a cellular telephone network, a wireless network, a Wi-Fi® network,another type of network, or a combination of two or more such networks.For example, the network 106 or a portion of the network 106 may includea wireless or cellular network and the connection to the network 106 maybe a Code Division Multiple Access (CDMA) connection, a Global Systemfor Mobile communications (GSM) connection, or another type of cellularor wireless coupling. In this example, the coupling may implement any ofa variety of types of data transfer technology, such as Single CarrierRadio Transmission Technology (1×RTT), Evolution-Data Optimized (EVDO)technology, General Packet Radio Service (GPRS) technology, EnhancedData rates for GSM Evolution (EDGE) technology, third-GenerationPartnership Project (3GPP) including 3G, fourth-generation wireless (4G)networks, Universal Mobile Telecommunications System (UMTS), High-SpeedPacket Access (HSPA), Worldwide Interoperability for Microwave Access(WiMAX), Long-Term Evolution (LTE) standard, or others defined byvarious standard-setting organizations, other long-range protocols, orother data transfer technology.

The messaging server system 108 provides server-side functionality viathe network 106 to a particular messaging client application 104. Whilecertain functions of the messaging system 100 are described herein asbeing performed by either a messaging client application 104 or by themessaging server system 108, it will be appreciated that the location ofcertain functionality either within the messaging client application 104or the messaging server system 108 is a design choice. For example, itmay be technically preferable to initially deploy certain technology andfunctionality within the messaging server system 108, but to latermigrate this technology and functionality to the messaging clientapplication 104 where a client device 102 has a sufficient processingcapacity.

The messaging server system 108 supports various services and operationsthat are provided to the messaging client application 104. Suchoperations include transmitting data to, receiving data from, andprocessing data generated by the messaging client application 104. Thisdata may include message content, client device information, geolocationinformation, media annotation and overlays, message content persistenceconditions, social network information, and live event information, asexamples. Data exchanges within the messaging system 100 are invoked andcontrolled through functions available via user interfaces (UIs) of themessaging client application 104.

Turning now specifically to the messaging server system 108, anApplication Programming Interface (API) server 110 is coupled to, andprovides a programmatic interface to, an application server 112. Theapplication server 112 is communicatively coupled to a database server118, which facilitates access to a database 120 in which is stored dataassociated with messages processed by the application server 112.

The API server 110 receives and transmits message data (e.g., commandsand message payloads) between the client device 102 and the applicationserver 112. Specifically, the API server 110 provides a set ofinterfaces (e.g., routines and protocols) that can be called or queriedby the messaging client application 104 in order to invoke functionalityof the application server 112. The API server 110 exposes variousfunctions supported by the application server 112, including accountregistration; login functionality; the sending of messages, via theapplication server 112, from a particular messaging client application104 to another messaging client application 104; the sending of mediafiles (e.g., images or video) from a messaging client application 104 tothe application server 112, for possible access by another messagingclient application 104; the setting of a collection of media data (e.g.,story); the retrieval of a list of friends of a user of a client device102; the retrieval of such collections; the retrieval of messages andcontent; the adding and deletion of friends to and from a social graph;the location of friends within a social graph; and the detecting of anapplication event (e.g., relating to the messaging client application104).

The application server 112 hosts a number of applications andsubsystems, including a messaging server application 114 and a socialnetwork system 116. The messaging server application 114 implements anumber of message processing technologies and functions, particularlyrelated to the aggregation and other processing of content (e.g.,textual and multimedia content) included in messages received frommultiple instances of the messaging client application 104. As will bedescribed in further detail, the text and media content from multiplesources may be aggregated into collections of content (e.g., calledstories or galleries). These collections are then made available, by themessaging server application 114, to the messaging client application104. Other processor- and memory-intensive processing of data may alsobe performed server-side by the messaging server application 114, inview of the hardware requirements for such processing.

The social network system 116 supports various social networkingfunctions and services, and makes these functions and services availableto the messaging server application 114. To this end, the social networksystem 116 maintains and accesses an entity graph within the database120. Examples of functions and services supported by the social networksystem 116 include the identification of other users of the messagingsystem 100 with whom a particular user has relationships or whom theuser is “following,” and also the identification of other entities andinterests of a particular user.

FIG. 2 is block diagram illustrating further details regarding themessaging system 100, according to exemplary embodiments. Specifically,the messaging system 100 is shown to comprise the messaging clientapplication 104 and the application server 112, which in turn embody anumber of subsystems, namely an ephemeral timer system 202, a collectionmanagement system 204, an annotation system 206, and an image processingsystem 208.

The ephemeral timer system 202 is responsible for enforcing thetemporary access to content permitted by the messaging clientapplication 104 and the messaging server application 114. To this end,the ephemeral timer system 202 incorporates a number of timers that,based on duration and display parameters associated with a message, orcollection of messages (e.g., a SNAPCHAT story), selectively display andenable access to messages and associated content via the messagingclient application 104. Further details regarding the operation of theephemeral timer system 202 are provided below.

The collection management system 204 is responsible for managingcollections of media (e.g., collections of text, image, video, and audiodata). In some examples, a collection of content (e.g., messages,including images, video, text, and audio) may be organized into an“event gallery” or an “event story.” Such a collection may be madeavailable for a specified time period, such as the duration of an eventto which the content relates. For example, content relating to a musicconcert may be made available as a “story” for the duration of thatmusic concert. The collection management system 204 may also beresponsible for publishing an icon that provides notification of theexistence of a particular collection to the user interface of themessaging client application 104.

The collection management system 204 furthermore includes a curationinterface 210 that allows a collection manager to manage and curate aparticular collection of content. For example, the curation interface210 enables an event organizer to curate a collection of contentrelating to a specific event (e.g., delete inappropriate content orredundant messages). Additionally, the collection management system 204employs machine vision (or image recognition technology) and contentrules to automatically curate a content collection. In certainembodiments, compensation may be paid to a user for inclusion ofuser-generated content in a collection. In such cases, the curationinterface 210 operates to automatically make payments to such users forthe use of their content.

The annotation system 206 provides various functions that enable a userto annotate or otherwise modify or edit media content associated with amessage. For example, the annotation system 206 provides functionsrelated to the generation and publishing of media overlays for messagesprocessed by the messaging system 100. For example, the annotationsystem 206 operatively supplies a media overlay (e.g., a SNAPCHATfilter) to the messaging client application 104 based on a geolocationof the client device 102. In another example, the annotation system 206operatively supplies a media overlay to the messaging client application104 based on other information, such as social network information ofthe user of the client device 102. A media overlay may include audio andvisual content and visual effects. Examples of audio and visual contentinclude pictures, texts, logos, animations, and sound effects. Anexample of a visual effect includes color overlaying. The audio andvisual content or the visual effects can be applied to a media contentitem (e.g., a photo) at the client device 102. For example, the mediaoverlay may include text that can be overlaid on top of a photographgenerated by the client device 102. In another example, the mediaoverlay includes an identification of a location (e.g., Venice Beach), aname of a live event, or a name of a merchant (e.g., Beach CoffeeHouse). In another example, the annotation system 206 uses thegeolocation of the client device 102 to identify a media overlay thatincludes the name of a merchant at the geolocation of the client device102. The media overlay may include other indicia associated with themerchant. The media overlays may be stored in the database 120 andaccessed through the database server 118.

In one exemplary embodiment, the annotation system 206 provides auser-based publication platform that enables users to select ageolocation on a map, and upload content associated with the selectedgeolocation. The user may also specify circumstances under which aparticular media overlay should be offered to other users. Theannotation system 206 generates a media overlay that includes theuploaded content and associates the uploaded content with the selectedgeolocation.

In another exemplary embodiment, the annotation system 206 provides amerchant-based publication platform that enables merchants to select aparticular media overlay associated with a geolocation via a biddingprocess. For example, the annotation system 206 associates the mediaoverlay of a highest-bidding merchant with a corresponding geolocationfor a predefined amount of time.

The image processing system 208 is dedicated to performing various imageprocessing operations, in some instances, with respect to images orvideo received within the payload of a message at the messaging serverapplication 114. As an example, the image processing system 208 providesfunctionality to allow a user to select an object or other element in anoriginal image to be removed and replaced using other portions of theimage. Further details regarding the image processing system 208 arediscussed below in reference to FIG. 4 , according to some embodiments.

FIG. 3 is a schematic diagram 300 illustrating data which may be storedin the database 120 of the messaging server system 108, according tocertain exemplary embodiments. While the content of the database 120 isshown to comprise a number of tables, it will be appreciated that thedata could be stored in other types of data structures (e.g., as anobject-oriented database).

The database 120 includes message data stored within a message table314. An entity table 302 stores entity data, including an entity graph304. Entities for which records are maintained within the entity table302 may include individuals, corporate entities, organizations, objects,places, events, etc. Regardless of type, any entity regarding which themessaging server system 108 stores data may be a recognized entity. Eachentity is provided with a unique identifier, as well as an entity typeidentifier (not shown).

The entity graph 304 furthermore stores information regardingrelationships and associations between or among entities. Suchrelationships may be social, professional (e.g., work at a commoncorporation or organization), interested-based, or activity-based,merely for example.

The database 120 also stores annotation data, in the example form offilters, in an annotation table 312. Filters for which data is storedwithin the annotation table 312 are associated with and applied tovideos (for which data is stored in a video table 310) and/or images(for which data is stored in an image table 308). Filters, in oneexample, are overlays that are displayed as overlaid on an image orvideo during presentation to a recipient user. Filters may be of variestypes, including user-selected filters from a gallery of filterspresented to a sending user by the messaging client application 104 whenthe sending user is composing a message. Other types of filters includegeolocation filters (also known as geo-filters), which may be presentedto a sending user based on geographic location. For example, geolocationfilters specific to a neighborhood or special location may be presentedwithin a user interface by the messaging client application 104, basedon geolocation information determined by a Global Positioning System(GPS) unit of the client device 102. Another type of filter is a datafilter, which may be selectively presented to a sending user by themessaging client application 104, based on other inputs or informationgathered by the client device 102 during the message creation process.Examples of data filters include a current temperature at a specificlocation, a current speed at which a sending user is traveling, abattery life for a client device 102, or the current time.

Other annotation data that may be stored within the image table 308 isso-called “lens” data. A “lens” may be a real-time special effect andsound that may be added to an image or a video.

As mentioned above, the video table 310 stores video data which, in oneembodiment, is associated with messages for which records are maintainedwithin the message table 314. Similarly, the image table 308 storesimage data associated with messages for which message data is stored inthe entity table 302. The entity table 302 may associate variousannotations from the annotation table 312 with various images and videosstored in the image table 308 and the video table 310.

A story table 306 stores data regarding collections of messages andassociated image, video, or audio data, which are compiled into acollection (e.g., a SNAPCHAT story or a gallery). The creation of aparticular collection may be initiated by a particular user (e.g., auser for whom a record is maintained in the entity table 302). A usermay create a “personal story” in the form of a collection of contentthat has been created and sent/broadcast by that user. To this end, theuser interface of the messaging client application 104 may include anicon that is user-selectable to enable a sending user to add specificcontent to his or her personal story.

A collection may also constitute a “live story,” which is a collectionof content from multiple users that is created manually, automatically,or using a combination of manual and automatic techniques. For example,a “live story” may constitute a curated stream of user-submitted contentfrom various locations and events. Users whose client devices havelocation services enabled and who are at a common location or event at aparticular time may, for example, be presented with an option, via auser interface of the messaging client application 104, to contributecontent to a particular live story. The live story may be identified tothe user by the messaging client application 104, based on his or herlocation. The end result is a “live story” told from a communityperspective.

A further type of content collection is known as a “location story,”which enables a user whose client device 102 is located within aspecific geographic location (e.g., on a college or university campus)to contribute to a particular collection. In some embodiments, acontribution to a location story may require a second degree ofauthentication to verify that the end user belongs to a specificorganization or other entity (e.g., is a student on the universitycampus).

FIG. 4 is a block diagram illustrating functional components of theimage processing system 208 that forms part of the messaging system 100,according to some example embodiments. To avoid obscuring the inventivesubject matter with unnecessary detail, various functional components(e.g., modules, engines, and databases) that are not germane toconveying an understanding of the inventive subject matter have beenomitted from FIG. 4 . However, a skilled artisan will readily recognizethat various additional functional components may be supported by theimage processing system 208 to facilitate additional functionality thatis not specifically described herein. As shown, the image processingsystem 208 includes a preprocessing component 402, an inpaintingcomponent 404, and a smoothing component 406. The above referencedfunctional components of the image processing system 208 are configuredto communicate with each other (e.g., via a bus, shared memory, aswitch, or APIs). Collectively, these components facilitate inpaintingof a user-selected region of an image using local patch matches in theimage. In other words, the preprocessing component 402, inpaintingcomponent 404, and smoothing component 406 work in conjunction to allowa user to select an object or other element in an original image to beremoved and replaced using other portions of the image, thereby makingthe resulting modified image without the object or other element appearnatural.

The preprocessing component 402 is responsible for performing varioustransformations to images prior to inpainting to improve (e.g.,optimize) runtime speed of the inpainting. To this end, thepreprocessing component 402 is configured to determine a local region inan image that serves as a boundary to limit the computations involved inthe inpainting process to a neighboring region surrounding theuser-selected region of the image. A size of the local region isdynamically computed based on a size of the user-selected region. Thepreprocessing component 402 is further configured to pad the height andwidth dimensions of the image by a predetermined padding size toincorporate enough background to calculate patch match statistics whenthe user-selected region is close to the image border. Additionally, thepreprocessing component 402 may further enlarge the local region by thepredetermined padding size, and scale (e.g., resize) the user-selectedregion to a predetermined size (e.g., 100×75 pixels) as part of theruntime speed optimization.

The inpainting component 404 is configured to inpaint the user-selectedregion using local patch match statistics. As part of this process, theinpainting component 404 identifies patch matches (e.g., identicalgroupings of pixels) in the local region and obtains the offsets of thepatch matches (e.g., a distance between patch matches defined bytwo-dimensional coordinates). The inpainting component 404 inpaints theuser-selected region using at least a portion of the patch matches fromthe local region. More specifically, the inpainting component 404 fillsthe user-selected region by combining a stack of shift images based onpatch offset statistics computed for the patch matches in the localregion. The inpainting of the user-selected region in the original imageresults in a modified image that appears natural despite omitting whatwas previously shown in the user-selected region.

The smoothing component 406 is configured to blend the inpainted regioninto the image and smooth any strong edges (e.g., pronounced imagebrightness discontinuities) resulting from the inpainting process. Forconventional methodologies, the final image blending step can be a speedbottleneck, especially in mobile configurations and/or when theuser-selected region is large. To improve upon conventionalmethodologies, the smoothing component 406 may apply fast andlightweight blending techniques such as Laplacian blending to theinpainted region to optimize runtime speed. Additionally, as notedabove, the inpainting process may, in some instances, introduce strongedges into the resulting modified image. To smooth these strong edges,the smoothing component 406 applies edge-preserving filtering to theinitial inpainting result (e.g., the modified image produced by theinpainting component 404) to blur insignificant edges, and the smoothingcomponent 406 then identifies strong edges in the resulting filteredgrayscale image to produce an edge map. The smoothing component 406 mayfurther dilate the resulting edge map to generate a binary mask forpossible strong edge pixels. The smoothing component 406 also appliesblurring techniques (e.g., Gaussian blur) on the inpainting result togenerate a blurred version of the image, which is blended together withthe initial inpainting result by applying blending techniques (e.g.,Laplacian blending) with the binary mask mentioned above.

As is understood by skilled artisans in the relevant computer andInternet-related arts, each functional component illustrated in FIG. 4may be implemented using hardware (e.g., a processor of a machine) or acombination of logic (e.g., executable software instructions) andhardware (e.g., memory and the processor of a machine) for executing thelogic. For example, any component included as part of the imageprocessing system 208 may physically include an arrangement of one ormore processors 408 (e.g., a subset of or among one or more processorsof a machine) configured to perform the operations described herein forthat component. As another example, any component of the imageprocessing system 208 may include software, hardware, or both, thatconfigure an arrangement of the one or more processors 408 to performthe operations described herein for that component. Accordingly,different components of the image processing system 208 may include andconfigure different arrangements of such processors 408 or a singlearrangement of such processors 408 at different points in time.

Furthermore, the various functional components depicted in FIG. 4 mayreside on a single machine (e.g., a client device or a server) or may bedistributed across several machines in various arrangements such ascloud-based architectures. Moreover, any two or more of these componentsmay be combined into a single component, and the functions describedherein for a single component may be subdivided among multiplecomponents. Functional details of these components are described belowwith respect to FIGS. 5-8 .

FIGS. 5-8 are flow charts illustrating operations of the imageprocessing system 208 in performing an example method 500 for digitalimage editing, according to some embodiments. The method 500 may beembodied in computer-readable instructions for execution by one or moreprocessors such that the operations of the method 500 may be performedin part or in whole by the functional components of the image processingsystem 208; accordingly, the method 500 is described below by way ofexample with reference thereto. However, it shall be appreciated that atleast some of the operations of the method 500 may be deployed onvarious other hardware configurations and the method 500 is not intendedto be limited to the image processing system 208.

In the context of method 500, the image processing system 208 accessesan image stored on the client device 102 or at the messaging serversystem 108. The image may be displayed within or as part of a userinterface provided by the messaging client application 104 forpresentation on the client device 102, and in some instances, the imagemay be captured by the client device 102.

At operation 505, the image processing system 208 receives a user inputidentifying a user-selected region of the image. The user may providethe user input identifying the selected region of the image by tracing aborder of the region on the image by way of appropriate interaction withan input device of the client device 102 (e.g., using a finger to tracethe border on a touch screen of the client device 102). Accordingly, theuser-selected region of the image is not limited to any particular shapeor size. For purposes of clarity in describing the method 500, the imageon which the user selects the region may be referred to as the “originalimage.”

At operation 510, the preprocessing component 402 determines a localregion for the user-selected region of the original image based on asize of the user-selected region. The local region includes a portion ofthe original image outside of the user-selected region and thatsurrounds the user-selected region. The determining of the local regionincludes dynamically computing a size (e.g., height and width) of thelocal region based on a size of the user-selected region. For example,if the height of the user-selected region is h and the width of theuser-selected region is w, the preprocessing component 402 may computethe height of the local region to be 2 h and the width to be 2 w.

At operation 515, the preprocessing component 402 enlarges the localregion of the original image by a predefined padding size. For example,the preprocessing component 402 may enlarge the height, h, of the localregion by h/2, and the width, w, by w/2.

At operation 520, the preprocessing component 402 scales (e.g., resizes)the user-selected region of the original image to a predetermined size.For example, the preprocessing component 402 may down scale theuser-selected region by resizing it to a predetermined size of 100pixels×75 pixels. The scaling of the user-selected region yields ascaled user-selected region.

At operation 525, the preprocessing component 402 pads the originalimage by the predefined padding size. In padding the original image, thepreprocessing component 402 copies an outer boundary of the originalimage, and appends the copied outer boundary to the original imageborder such that the appended copy of the outer boundary creates amirrored reflection of the actual outer boundary. As an example, thepreprocessing component 402 may pad the height, h, of the original imageby h/2, and the width, w, by w/2.

At operation 530, the inpainting component 404 computes a binary maskfor the scaled user-selected region. In computing the binary mask, theinpainting component 404 marks pixels inside the scaled user-selectedregion as “1,” and marks pixels in the remainder of the image as “0.”

At operation 535, the inpainting component 404 identifies patch matcheswithin the enlarged local region. Each patch match comprises twoidentical image patches, and each image patch comprises one or morepixels of the original image. By limiting the search for patch matchesto the enlarged local region rather than the entire image, theinpainting component 404 may achieve a faster runtime speed.

To identify patch matches within the enlarged local region of theoriginal image, the inpainting component 404 may apply a PatchMatchalgorithm to the enlarged local region. The PatchMatch algorithm findsthe patch matches by defining a nearest-neighbor field (NNF) as afunction ƒ: R²→R² of offsets, which is over all possible patch matchesin the enlarged local region, for some distance function D between twopatches. The algorithm comprises three main operations: 1) fill the NNFwith either random offsets or some prior information; 2) apply aniterative update process to the NNF, in which good patch offsets arepropagated to adjacent pixels; and 3) perform a random search in theneighborhood of the best offset found so far. Independently of thesethree operations, the PatchMatch algorithm may also use a coarse-to-fineapproach by building an image pyramid to obtain a better result.

At operation 540, the inpainting component 404 inpaints (e.g., fills)the masked region (e.g., the scaled user-selected region) in theoriginal image using a portion of the identified patch matches. Theinpainting of the masked region in the original image results in amodified image where the user-selected region has been filled with otherportions of the image to produce an image that appears natural despiteomitting what was previously shown in the user-selected region.

At operation 545, the smoothing component 406 blends the inpaintedregion into the modified image. For example, the smoothing component 406may apply Laplacian blending to blend the inpainted region into themodified image. The blending of the inpainted region results in amodified image that appears even more natural than the initial inpaintedresult produced as a result of operation 540.

As shown in FIG. 6 , the method 500 may, in some embodiments, alsoinclude operations 605, 610, 615, and 620. The operations 605 and 610may be performed subsequent to operation 535, in which the inpaintingcomponent 404 identifies patch matches within the enlarged local region,or as part of the operation 540, in which the inpainting component 404inpaints the user-selected region in the original image using a portionof the patch match offsets. At operation 605, the inpainting component404 computes patch offsets for the patch matches identified, atoperation 535, within the enlarged local region. A patch offset includestwo-dimensional coordinates representing a distance between two patchmatches. As an example, for each patch, P, in the enlarged local region,the inpainting component 404 computes its offset s to its most similarpatch according to the following function:s(x)=arg min ∥P(x+s)−P(x)H∥ ² s.t. |s|>τ.Here, s=(u, v) is the two-dimensional coordinates of the offset, x=(x,y) is the position of a patch, and P(x) is the patch centered betweentwo patches. The threshold τ is to preclude nearby patches.

At operation 610, the inpainting component 404 determines patch offsetstatistics based on the patch offsets. For example, given all theoffsets s(x), the inpainting component 404 computes their statistics bya two-dimensional histogram h(u, v):h(u,v)=Σ_(x)δ(s(x)=(u,v)).Here, δ(.) is 1 when the argument is true and 0 otherwise.

Operation 615 may be performed as part of the operation 540, in whichthe inpainting component 404 inpaints the user-selected region using aportion of the identified patch matches. At operation 615, theinpainting component 404 combines shifted images based on the offsetstatistics to fill the scaled user-selected region to produce themodified image. The combining of the shifted images may include applyingthe pixel-level graph cut algorithm, where the inpainting component 404applies a label to each pixel in the enlarged local region thatcorresponds to a possible offset in the histogram while also enforcingpairwise consistency constraints between two neighboring pixels.

For example, the inpainting component 404 may identify the K dominantoffsets in the two-dimensional histogram, which are the K highest peaksin the histogram. Given the K offsets, the inpainting component 404optimizes the following Markov random field (MRF) energy function:

$\begin{matrix}{{E(L)} = {{\sum\limits_{x \in \Omega}{E_{d}( {L(x)} )}} + {\sum\limits_{{{({x,x^{\prime}})}|{x \in \Omega}},{x^{\prime} \in \Omega}}{E_{s{({{L{(x)}},{L{(x^{\prime})}}})}}.}}}} & \;\end{matrix}$Here, Q is the user-selected region, and (x, x′) are 4-connectedneighbors. L is a label representing the pre-selected offsets{s_(i)}^(K) _(i=1) or s₀=(0,0)¹. “L(x)=i” means that the inpaintingcomponent 404 copies the pixel at x+5, to the location x. The term E_(d)is 0 if the label is valid (x+s_(i) is a known pixel), and if not it is+∞. The smoothness term E_(s) penalizes incoherent seams. When a=L(x)and b=L(x′), E_(s) is defined as:E _(s)(a,b)=∥(x+s _(a))−I(x+s _(b))∥² +∥I(x′+s _(a))−I(x′+s _(b))∥².Here, I(x) is the RGB color of x. I(x+s) is a shifted image given fixeds. If s_(a)≠s_(b) there is a seam between x and x′. Thus, the aboveequation penalizes such a seam that the two shifted images I(x+s_(a))and I(x+s_(b)) are not similar near this seam.

Operation 620 may be performed subsequent to operation 540, in which theinpainting component 404 inpaints the user selected region in theoriginal image using a portion of the identified patch matches. Atoperation 620, the inpainting component 404 upscales the inpaintedregion to the original image resolution. For example, the inpaintingcomponent 404 may employ a super resolution step using the graph cutalgorithm. To illustrate this example, the following function may beused:E(X)=Σ_(i)ϕ(x _(i))+Σ_(j in N(xi))Ψ(x _(i) ,x _(j))Here, x is a set of known pixels and N(x_(i)) is a neighborhood ofunknown pixels x_(i) after applying transform to its closest inpaintedpixel. For each unknown pixel (x_(i), y_(i)), the inpainting componentfinds the closest inpainted pixel (x₀, y₀) and then applies a (x, y)transform to get the new pixel location (x₀+x, y₀+y), then find a4-pixel neighborhood of the pixel location (x₀+x−(x_(i)−x₀),y₀+y−(y_(i)−y₀)). The graph cut algorithm then assigns one of these fourpixel values to (x_(i), y_(i)), based on color intensity matching scoreas well as a neighboring pixel consistency constraint. To furtheroptimize the runtime speed on mobile devices, the image processingsystem 208 may constrain the search neighborhood such that each pixel inthe low-resolution result can only map back to its top left or bottomright neighbors (rather than all four neighboring pixels).

As shown in FIG. 7 , the method 500 may also include operations 705,710, 715, 720, 725, 730, and 735, which may be performed subsequently tothe operation 535, in which the inpainting component 404 identifiespatch matches within the enlarged local region. At operation 705, theinpainting component 404 computes patch offset statistics in the mannerdescribed above with reference to operation 610.

At operation 710, the inpainting component 404 determines a mean andstandard deviation of the offsets in the histogram described above inreference to operation 610. At operation 715, the inpainting component404 determines a ratio between the mean and standard deviation. Atoperation 720, the inpainting component 404 determines whether the ratiois above a predefined threshold. If, at operation 720, the inpaintingcomponent 404 determines that the ratio is above the threshold, themethod 500 proceeds to operation 540, where the inpainting component 404inpaints the user-selected region in the original image using a portionof the identified patch matches.

If, at operation 720, the inpainting component 404 determines that theratio is not above the threshold, the method 500 proceeds to operation725, where the inpainting component 404 computes a mean color ofneighboring pixels (e.g., pixels near the user-selected region). Atoperation 730, the inpainting component 404 generates a color mask usingthe mean color of the neighboring pixels. At operation 735, theinpainting component 404 fills the user-selected region using the colormask.

As shown in FIG. 8 , the method 500 may, in some embodiments, includeoperations 805, 810, 815, 820, and 825. Operations 805, 810, 815, and820 may be performed subsequently to operation 540, in which theinpainting component 404 inpaints the user-selected region using aportion of the identified patch matches, thereby generating the modifiedimage. At operation 805, the smoothing component 406 identifies one ormore strong edges in the modified image. The smoothing component 406 mayidentify the strong edges by applying edge-preserving filtering to themodified image to blur insignificant edges in the modified image. Theapplication of the edge-preserving filtering results in a grayscaleimage.

At operation 810, the smoothing component 406 generates an edge map forthe modified image based on the grayscale image. At operation 815, thesmoothing component 406 generates a binary mask for possible strongedges in the modified image. The smoothing component 406 may generatethe binary mask by dilating the edge map.

At operation 820, the smoothing component 406 applies blurringtechniques to the modified image to produce a blurred version of themodified image (referred to hereinafter as the “blurred image”). Forexample, the smoothing component 406 may apply Gaussian blurring to themodified image to produce the blurred image.

Operation 825 may be performed in parallel with or as part of operation545, in which the smoothing component 406 blends the inpainted regioninto the modified image. At operation 825, the smoothing component 406blends the modified image with the blurred image using the binary maskto produce an further modified image with no strong edges. In blendingthe modified image with the blurred image, the smoothing component 406may apply Laplacian blending.

FIGS. 9A and 9B are interface diagrams illustrating aspects of userinterfaces provided by the messaging system, according to someembodiments. In particular, FIG. 9A illustrates an original image 900that may be captured by and presented within a user interface display onthe client device 102. The original image 900 includes a user-selectedregion 902. As an example, a user of the client device 102 may selectthe region 902 by using his or her finger to trace an outline of theregion 902 on a touch screen of the client device 102, although anyother appropriate input device (e.g., a mouse) may be used to trace anoutline of the region 902. An object, specifically a sign, is shownwithin the user-selected region 902.

FIG. 9B illustrates a modified image 950 that may be presented within auser interface display on the client device 102. The modified image 950is an edited version of the original image 900 generated by applying thetechniques described herein to the user-selected region 902. Morespecifically, in the modified image 950 the user-selected region 902 hasbeen removed and replaced with other portions of the original image 900to create a natural-looking image without the object shown within theuser-selected region 902 of the original image 900.

Software Architecture

FIG. 10 is a block diagram illustrating an example software architecture1006, which may be used in conjunction with various hardwarearchitectures herein described. FIG. 10 is a non-limiting example of asoftware architecture and it will be appreciated that many otherarchitectures may be implemented to facilitate the functionalitydescribed herein. The software architecture 1006 may execute on hardwaresuch as a machine 1100 of FIG. 11 that includes, among other things,processors 1104, memory/storage 1106, and I/O components 1118. Arepresentative hardware layer 1052 is illustrated and can represent, forexample, the machine 1100 of FIG. 11 . The representative hardware layer1052 includes a processing unit 1054 having associated executableinstructions 1004. The executable instructions 1004 represent theexecutable instructions of the software architecture 1006, includingimplementation of the methods, components, and so forth describedherein. The hardware layer 1052 also includes memory and/or storagemodules memory/storage 1056, which also have the executable instructions1004. The hardware layer 1052 may also comprise other hardware 1058.

As used herein, the term “component” may refer to a device, a physicalentity, or logic having boundaries defined by function or subroutinecalls, branch points, APIs, and/or other technologies that provide forthe partitioning or modularization of particular processing or controlfunctions. Components may be combined via their interfaces with othercomponents to carry out a machine process. A component may be a packagedfunctional hardware unit designed for use with other components and apart of a program that usually performs a particular function of relatedfunctions.

Components may constitute either software components (e.g., codeembodied on a machine-readable medium) or hardware components. A“hardware component” is a tangible unit capable of performing certainoperations and may be configured or arranged in a certain physicalmanner. In various exemplary embodiments, one or more computer systems(e.g., a standalone computer system, a client computer system, or aserver computer system) or one or more hardware components of a computersystem (e.g., a processor or a group of processors) may be configured bysoftware (e.g., an application or application portion) as a hardwarecomponent that operates to perform certain operations as describedherein. A hardware component may also be implemented mechanically,electronically, or any suitable combination thereof. For example, ahardware component may include dedicated circuitry or logic that ispermanently configured to perform certain operations.

A hardware component may be a special-purpose processor, such as aField-Programmable Gate Array (FPGA) or an Application-SpecificIntegrated Circuit (ASIC). A hardware component may also includeprogrammable logic or circuitry that is temporarily configured bysoftware to perform certain operations. For example, a hardwarecomponent may include software executed by a general-purpose processoror other programmable processor. Once configured by such software,hardware components become specific machines (or specific components ofa machine) uniquely tailored to perform the configured functions and areno longer general-purpose processors. It will be appreciated that thedecision to implement a hardware component mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

A processor may be, or include, any circuit or virtual circuit (aphysical circuit emulated by logic executing on an actual processor)that manipulates data values according to control signals (e.g.,“commands,” “op codes,” “machine code,” etc.) and that producescorresponding output signals that are applied to operate a machine. Aprocessor may, for example, be a Central Processing Unit (CPU), aReduced Instruction Set Computing (RISC) processor, a ComplexInstruction Set Computing (CISC) processor, a Graphics Processing Unit(GPU), a Digital Signal Processor (DSP), an ASIC, a Radio-FrequencyIntegrated Circuit (RFIC), or any combination thereof. A processor mayfurther be a multi-core processor having two or more independentprocessors (sometimes referred to as “cores”) that may executeinstructions contemporaneously.

Accordingly, the phrase “hardware component” (or “hardware-implementedcomponent”) should be understood to encompass a tangible entity, be thatan entity that is physically constructed, permanently configured (e.g.,hardwired), or temporarily configured (e.g., programmed) to operate in acertain manner or to perform certain operations described herein.Considering embodiments in which hardware components are temporarilyconfigured (e.g., programmed), each of the hardware components need notbe configured or instantiated at any one instance in time. For example,where a hardware component comprises a general-purpose processorconfigured by software to become a special-purpose processor, thegeneral-purpose processor may be configured as respectively differentspecial-purpose processors (e.g., comprising different hardwarecomponents) at different times. Software accordingly configures aparticular processor or processors, for example, to constitute aparticular hardware component at one instance of time and to constitutea different hardware component at a different instance of time. Hardwarecomponents can provide information to, and receive information from,other hardware components. Accordingly, the described hardwarecomponents may be regarded as being communicatively coupled. Wheremultiple hardware components exist contemporaneously, communications maybe achieved through signal transmission (e.g., over appropriate circuitsand buses) between or among two or more of the hardware components. Inembodiments in which multiple hardware components are configured orinstantiated at different times, communications between or among suchhardware components may be achieved, for example, through the storageand retrieval of information in memory structures to which the multiplehardware components have access.

For example, one hardware component may perform an operation and storethe output of that operation in a memory device to which it iscommunicatively coupled. A further hardware component may then, at alater time, access the memory device to retrieve and process the storedoutput. Hardware components may also initiate communications with inputor output devices, and can operate on a resource (e.g., a collection ofinformation). The various operations of example methods described hereinmay be performed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implementedcomponents that operate to perform one or more operations or functionsdescribed herein. As used herein, “processor-implemented component”refers to a hardware component implemented using one or more processors.Similarly, the methods described herein may be at least partiallyprocessor-implemented, with a particular processor or processors beingan example of hardware. For example, at least some of the operations ofa method may be performed by one or more processors orprocessor-implemented components.

Moreover, the one or more processors may also operate to supportperformance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by a group of computers(as examples of machines including processors), with these operationsbeing accessible via a network (e.g., the Internet) and via one or moreappropriate interfaces (e.g., an API). The performance of certain of theoperations may be distributed among the processors, not only residingwithin a single machine, but deployed across a number of machines. Insome exemplary embodiments, the processors or processor-implementedcomponents may be located in a single geographic location (e.g., withina home environment, an office environment, or a server farm). In otherexemplary embodiments, the processors or processor-implementedcomponents may be distributed across a number of geographic locations.

In the exemplary architecture of FIG. 10 , the software architecture1006 may be conceptualized as a stack of layers where each layerprovides particular functionality. For example, the softwarearchitecture 1006 may include layers such as an operating system 1002,libraries 1020, frameworks/middleware 1018, applications 1016, and apresentation layer 1014. Operationally, the applications 1016 and/orother components within the layers may invoke API calls 1008 through thesoftware stack and receive a response as messages 1010. The layersillustrated are representative in nature and not all softwarearchitectures have all layers. For example, some mobile orspecial-purpose operating systems may not provide aframeworks/middleware 1018, while others may provide such a layer. Othersoftware architectures may include additional or different layers.

The operating system 1002 may manage hardware resources and providecommon services. The operating system 1002 may include, for example, akernel 1022, services 1024, and drivers 1026. The kernel 1022 may act asan abstraction layer between the hardware and the other software layers.For example, the kernel 1022 may be responsible for memory management,processor management (e.g., scheduling), component management,networking, security settings, and so on. The services 1024 may provideother common services for the other software layers. The drivers 1026are responsible for controlling or interfacing with the underlyinghardware. For instance, the drivers 1026 include display drivers, cameradrivers, Bluetooth® drivers, flash memory drivers, serial communicationdrivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers,audio drivers, power management drivers, and so forth depending on thehardware configuration.

The libraries 1020 provide a common infrastructure that is used by theapplications 1016 and/or other components and/or layers. The libraries1020 provide functionality that allows other software components toperform tasks in an easier fashion than by interfacing directly with theunderlying operating system 1002 functionality (e.g., kernel 1022,services 1024, and/or drivers 1026). The libraries 1020 may includesystem libraries 1044 (e.g., C standard library) that may providefunctions such as memory allocation functions, string manipulationfunctions, mathematical functions, and the like. In addition, thelibraries 1020 may include API libraries 1046 such as media libraries(e.g., libraries to support presentation and manipulation of variousmedia formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG),graphics libraries (e.g., an OpenGL framework that may be used to render2D and 3D graphic content on a display), database libraries (e.g.,SQLite that may provide various relational database functions), weblibraries (e.g., WebKit that may provide web browsing functionality),and the like. The libraries 1020 may also include a wide variety ofother libraries 1048 to provide many other APIs to the applications 1016and other software components/modules.

The frameworks/middleware 1018 provide a higher-level commoninfrastructure that may be used by the applications 1016 and/or othersoftware components/modules. For example, the frameworks/middleware 1018may provide various graphic user interface (GUI) functions, high-levelresource management, high-level location services, and so forth. Theframeworks/middleware 1018 may provide a broad spectrum of other APIsthat may be utilized by the applications 1016 and/or other softwarecomponents/modules, some of which may be specific to a particularoperating system 1002 or platform.

The applications 1016 include built-in applications 1038 and/orthird-party applications 1040. Examples of representative built-inapplications 1038 may include, but are not limited to, a contactsapplication, a browser application, a book reader application, alocation application, a media application, a messaging application,and/or a game application. The third-party applications 1040 may includean application developed using the ANDROID™ or IOS™ software developmentkit (SDK) by an entity other than the vendor of the particular platform,and may be mobile software running on a mobile operating system such asIOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. Thethird-party applications 1040 may invoke the API calls 1008 provided bythe mobile operating system (such as the operating system 1002) tofacilitate functionality described herein.

The applications 1016 may use built-in operating system functions (e.g.,kernel 1022, services 1024, and/or drivers 1026), libraries 1020, andframeworks/middleware 1018 to create user interfaces to interact withusers of the system. Alternatively, or additionally, in some systemsinteractions with a user may occur through a presentation layer, such asthe presentation layer 1014. In these systems, the application/component“logic” can be separated from the aspects of the application/componentthat interact with a user.

Exemplary Machine

FIG. 11 is a block diagram illustrating components (also referred toherein as “modules”) of a machine 1100, according to some exemplaryembodiments, able to read instructions from a machine-readable medium(e.g., a machine-readable storage medium) and perform any one or more ofthe methodologies discussed herein. Specifically, FIG. 11 shows adiagrammatic representation of the machine 1100 in the example form of acomputer system, within which instructions 1110 (e.g., software, aprogram, an application, an applet, an app, or other executable code)for causing the machine 1100 to perform any one or more of themethodologies discussed herein may be executed. As such, theinstructions 1110 may be used to implement modules or componentsdescribed herein. The instructions 1110 transform the general,non-programmed machine 1100 into a particular machine 1100 programmed tocarry out the described and illustrated functions in the mannerdescribed. In alternative embodiments, the machine 1100 operates as astandalone device or may be coupled (e.g., networked) to other machines.In a networked deployment, the machine 1100 may operate in the capacityof a server machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 1100 may comprise, but not be limitedto, a server computer, a client computer, a personal computer (PC), atablet computer, a laptop computer, a netbook, a set-top box (STB), apersonal digital assistant (PDA), an entertainment media system, acellular telephone, a smart phone, a mobile device, a wearable device(e.g., a smart watch), a smart home device (e.g., a smart appliance),other smart devices, a web appliance, a network router, a networkswitch, a network bridge, or any machine capable of executing theinstructions 1110, sequentially or otherwise, that specify actions to betaken by machine 1100. Further, while only a single machine 1100 isillustrated, the term “machine” shall also be taken to include acollection of machines that individually or jointly execute theinstructions 1110 to perform any one or more of the methodologiesdiscussed herein.

The machine 1100 may include processors 1104, memory/storage 1106, andI/O components 1118, which may be configured to communicate with eachother such as via a bus 1102. The memory/storage 1106 may include amemory 1114, such as a main memory, or other memory storage, and astorage unit 1116, both accessible to the processors 1104 such as viathe bus 1102. The storage unit 1116 and memory 1114 store theinstructions 1110 embodying any one or more of the methodologies orfunctions described herein. The instructions 1110 may also reside,completely or partially, within the memory 1114, within the storage unit1116, within at least one of the processors 1104 (e.g., within theprocessor's cache memory), or any suitable combination thereof, duringexecution thereof by the machine 1100. Accordingly, the memory 1114, thestorage unit 1116, and the memory of the processors 1104 are examples ofmachine-readable media.

As used herein, the term “machine-readable medium,” “computer-readablemedium,” or the like may refer to any component, device, or othertangible medium able to store instructions and data temporarily orpermanently. Examples of such media may include, but are not limited to,random-access memory (RAM), read-only memory (ROM), buffer memory, flashmemory, optical media, magnetic media, cache memory, other types ofstorage (e.g., Electrically Erasable Programmable Read-Only Memory(EEPROM)), and/or any suitable combination thereof. The term“machine-readable medium” should be taken to include a single medium ormultiple media (e.g., a centralized or distributed database, orassociated caches and servers) able to store instructions. The term“machine-readable medium” may also be taken to include any medium, orcombination of multiple media, that is capable of storing instructions(e.g., code) for execution by a machine, such that the instructions,when executed by one or more processors of the machine, cause themachine to perform any one or more of the methodologies describedherein. Accordingly, a “machine-readable medium” may refer to a singlestorage apparatus or device, as well as “cloud-based” storage systems orstorage networks that include multiple storage apparatus or devices. Theterm “machine-readable medium” excludes signals per se.

The I/O components 1118 may include a wide variety of components toprovide a user interface for receiving input, providing output,producing output, transmitting information, exchanging information,capturing measurements, and so on. The specific I/O components 1118 thatare included in the user interface of a particular machine 1100 willdepend on the type of machine. For example, portable machines such asmobile phones will likely include a touch input device or other suchinput mechanisms, while a headless server machine will likely notinclude such a touch input device. It will be appreciated that the I/Ocomponents 1118 may include many other components that are not shown inFIG. 11 . The I/O components 1118 are grouped according to functionalitymerely for simplifying the following discussion and the grouping is inno way limiting. In various exemplary embodiments, the I/O components1118 may include output components 1126 and input components 1128. Theoutput components 1126 may include visual components (e.g., a displaysuch as a plasma display panel (PDP), a light emitting diode (LED)display, a liquid crystal display (LCD), a projector, or a cathode raytube (CRT)), acoustic components (e.g., speakers), haptic components(e.g., a vibratory motor, resistance mechanisms), other signalgenerators, and so forth. The input components 1128 may includealphanumeric input components (e.g., a keyboard, a touch screenconfigured to receive alphanumeric input, a photo-optical keyboard, orother alphanumeric input components), point-based input components(e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, orother pointing instruments), tactile input components (e.g., a physicalbutton, a touch screen that provides location and/or force of touches ortouch gestures, or other tactile input components), audio inputcomponents (e.g., a microphone), and the like. The input components 1128may also include one or more image-capturing devices, such as a digitalcamera for generating digital images and/or video.

In further exemplary embodiments, the I/O components 1118 may includebiometric components 1130, motion components 1134, environmentcomponents 1136, or position components 1138, as well as a wide array ofother components. For example, the biometric components 1130 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram-basedidentification), and the like. The motion components 1134 may includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environment components 1136 may include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detect concentrations of hazardous gases for safetyor to measure pollutants in the atmosphere), or other components thatmay provide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 1138 mayinclude location sensor components (e.g., a GPS receiver component),altitude sensor components (e.g., altimeters or barometers that detectair pressure from which altitude may be derived), orientation sensorcomponents (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 1118 may include communication components 1140operable to couple the machine 1100 to a network 1132 or devices 1120via a coupling 1124 and a coupling 1122 respectively. For example, thecommunication components 1140 may include a network interface componentor other suitable device to interface with the network 1132. In furtherexamples, the communication components 1140 may include wiredcommunication components, wireless communication components, cellularcommunication components, Near Field Communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components,and other communication components to provide communication via othermodalities. The devices 1120 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUSB).

Moreover, the communication components 1140 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 1140 may include Radio Frequency Identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF4111, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components1140, such as location via Internet Protocol (IP) geo-location, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

Where a phrase similar to “at least one of A, B, or C,” “at least one ofA, B, and C,” “one or more of A, B, or C,” or “one or more of A, B, andC” is used, it is intended that the phrase be interpreted to mean that Aalone may be present in an embodiment, B alone may be present in anembodiment, C alone may be present in an embodiment, or any combinationof the elements A, B, and C may be present in a single embodiment; forexample, A and B, A and C, B and C, or A and B and C may be present.

Changes and modifications may be made to the disclosed embodimentswithout departing from the scope of the present disclosure. These andother changes or modifications are intended to be included within thescope of the present disclosure, as expressed in the following claims.

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever. The following notice applies to the software and dataas described below and in the drawings that form a part of thisdocument: Copyright 2017, SNAPCHAT, INC., All Rights Reserved.

What is claimed is:
 1. A method comprising: identifying a selectedregion of an image, the selected region including a depicted object;generating a color mask using a mean color of pixels located in a localregion, the local region being a portion of the image and being outsideof the selected region; removing the depicted object from the image byreplacing the selected region with the color mask generated based on thelocal region in the image, the replacing of the selected region with thecolor mask yielding a modified image without the depicted object;identifying one or more strong edges in the modified image; generatingan edge map based on the identified one or more strong edges; generatinga binary mask for strong edge pixels by dilating the edge map; applyingblur techniques to the modified image to yield a blurred image; andgenerating a blended image by applying Laplacian blending with thebinary mask to the modified image and the blurred image.
 2. The methodof claim 1, wherein the identifying of the strong edges includesapplying edge-preserving filtering to the modified image to blurinsignificant edges in the modified image.
 3. The method of claim 1,wherein the blur techniques include Gaussian blur.
 4. The method ofclaim 1, wherein the operations comprise determining the local regionbased on a size of the selected region.
 5. The method of claim 1,wherein the determining the local region includes dynamically computingdimensions of the local region based on dimensions of the selectedregion, the method further comprising: responsive to detecting that theselected region is within a predetermined distance of a border of theimage: padding the height and width of the image based on apredetermined padding size; enlarging the local region by thepredetermined padding size; and scaling the selected region to apredetermined size.
 6. The method of claim 1, wherein the local regionsurrounds the selected region.
 7. The method of claim 1, wherein theoperations further comprise causing presentation of the modified imagewithout the depicted object on a display device.
 8. The method of claim1, wherein the selected region is identified based on an input from auser.
 9. The method in claim 1, wherein the blended image is overlaidwith a media overlay.
 10. The method in claim 1, further comprising: theblended image being sent by a messaging client application responsive todetecting a selection of a selectable icon displayed by a user interfaceof the messaging client application.
 11. A system comprising: aprocessor; and memory coupled to the processor and storing instructionsthat, when executed by the processor, cause the system to performoperations comprising: identifying a selected region of an image, theselected region including a depicted object; generating a color maskusing a mean color of pixels located in a local region, the local regionbeing a portion of the image and being outside of the selected region;and removing the depicted object from the image by replacing theselected region with the color mask, the replacing of the selectedregion with the color mask yielding a modified image without thedepicted object, wherein the operations further comprise: identifyingone or more strong edges in the modified image; generating an edge mapbased on the identified one or more strong edges; generating a binarymask for strong edge pixels by dilating the edge map; applying blurtechniques to the modified image to yield a blurred image; and blendingthe modified image with the blurred image using the binary mask.
 12. Thesystem of claim 11, wherein the identifying of the strong edges includesapplying edge-preserving filtering to the modified image to blurinsignificant edges in the modified image.
 13. The system of claim 11,wherein the blur techniques include Gaussian blur.
 14. The system ofclaim 11, wherein the blending of the modified image with the blurredimage includes applying Laplacian blending to the modified image and theblurred image with the binary mask.
 15. The system of claim 11, whereinthe operations comprise determining the local region based on a size ofthe selected region.
 16. The system of claim 11, wherein the determiningthe local region includes dynamically computing dimensions of the localregion based on dimensions of the selected region.
 17. The system ofclaim 11, wherein the local region surrounds the selected region. 18.The system of claim 11, wherein the operations further comprise causingpresentation of the modified image without the depicted object on adisplay device.
 19. A machine-readable non-transitory storage mediumhaving instruction data executable by a machine to cause the machine toperform operations comprising: identifying a selected region of animage, the selected region including a depicted object; generating acolor mask using a mean color of pixels located in a local region, thelocal region being a portion of the image and being outside of theselected region; and removing the depicted object from the image byreplacing the selected region with the color mask, the replacing of theselected region with the color mask yielding a modified image withoutthe depicted object; identifying one or more strong edges in themodified image; generating an edge map based on the identified one ormore strong edges; generating a binary mask for strong edge pixels bydilating the edge map; applying blur techniques to the modified image toyield a blurred image; and generating a blended image by applyingLaplacian blending with the binary mask to the modified image and theblurred image.